Translation display device, translation display method, and control program

ABSTRACT

A translation display device of the present invention carries out a translation process of translating text extracted from a certain image and displays translated text while a display frame rate is maintained. A translation processing section ( 5 ) carries out the translation process in a case where no image other than the certain image is being subjected to the translation process. An image movement analyzing section ( 7 ) identifies a displacement of a position of an object in a most recent image which has been most recently obtained, the displacement being measured with respect to a reference position of an object in a reference image for which the translation process most recently ended. A generated image in which translated text obtained by translating extracted text extracted from the reference image is displayed so as to be superimposed on the most recent image in accordance with (i) a position of the extracted text and (ii) information on the displacement.

TECHNICAL FIELD

The present invention relates to (i) a translation display device fortranslating text in an image and displaying the text thus translated,(ii) a method of displaying a translation, and (iii) a control program.

BACKGROUND ART

In recent years, a technique related to a translation display device hasbeen developed. Such a translation display device (i) extracts text froman image captured by a camera, (ii) identifies the text, (iii)translates the text into a predetermined language, and (iv) displays thetext thus translated.

For example, Patent Literature 1 discloses an information display devicefor (i) carrying out real-time translation or conversion of foreignlanguage text and/or a pictograph which has/have been extracted from animage by use of an optical character reader (OCR) and (ii) displayingtranslated or converted text and/or pictograph at a suitable displayposition. Patent Literature 2 discloses a technique for promptlydisplaying an image. According to the technique, (i) imagescorresponding to respective of a first frame and a second frame by whichthe first frame is immediately followed are analyzed, (ii) text whichhas been extracted and translated in the first frame is displayed in thesecond frame so that the text thus displayed follows a position of thetext in the first frame, and (iii) only text in a newly captured regionin the image of the second frame is extracted and translated.

CITATION LIST Patent Literature

Patent Literature 1

Japanese Patent Application Publication Tokukai No. 2011-134144(Publication date: Jul. 7, 2011)

Patent Literature 2

Japanese Patent Application Publication Tokukai No. 2010-103694(Publication date: May 6, 2010)

SUMMARY OF INVENTION Technical Problem

In connection with advancement in translation display devices, thenumber of extractable character types has increased and multilingualtranslation has become available. This has increased a length of timerequired for carrying out a text extracting process and a translationprocess, and therefore poses a problem for the conventional techniquesdisclosed in Patent Literatures 1 and 2. Specifically, according to theconventional techniques, a text in part of or an entire part of eachimage is extracted and translated. This unfortunately causes a reductionin display frame rate.

The present invention has been made in view of the problem, and anobject of the present invention is to provide (i) a translation displaydevice for translating text extracted from an image and displaying thetext thus translated while a frame rate is maintained, (ii) a method ofdisplaying translation, and (iii) a control program.

Solution to Problem

In order to attain the object, a translation display device inaccordance with an aspect of the present invention is a translationdisplay device including: an image obtaining section for sequentiallyobtaining images; a translating section for a carrying out a translationprocess of translating text extracted from any given image, thetranslating section subjecting a certain image to the translationprocess in a case where the image obtaining section obtains the certainimage while no image other than the certain image is being subjected tothe translation process; a movement information identifying section foridentifying movement information indicative of a displacement of aposition of an object in a most recent image which has been mostrecently obtained by the image obtaining section, the displacement beingmeasured with respect to a reference position of an object in areference image for which the translation process most recently ended;and a display section for displaying a generated image in whichtranslated text obtained by translating extracted text extracted fromthe reference image is superimposed on the most recent image inaccordance with (i) a position of the extracted text and (ii) themovement information.

A method of displaying translation in accordance with an aspect of thepresent invention is a method of displaying a translation, including thesteps of: (a) sequentially obtaining images; (b) carrying out atranslation process of translating text extracted from any given image,the step (b) being carried out so that a certain image is subjected tothe translation process in a case where the certain image is obtainedwhile no image other than the certain image is being subjected to thetranslation process; (c) identifying movement information indicative ofa displacement of a position of an object in a most recent image whichhas been most recently obtained in the step (a), the displacement beingmeasured with respect to a reference position of an object in areference image for which the translation process most recently ended inthe step (b); and (d) displaying a generated image in which translatedtext obtained by translating extracted text extracted from the referenceimage is superimposed on the most recent image in accordance with (i) aposition of the extracted text and (ii) the movement information.

Advantageous Effects of Invention

An aspect of the present invention brings about an effect of displaying,without delaying a timing at which an image is to be displayed (i.e.,without reducing a frame rate), an image in which a certain level ormore of accuracy is maintained in term of (i) a translation of text and(ii) a position at which translated text is to be displayed.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating main configurations of atranslation display device in accordance with Embodiment 1 of thepresent invention.

FIG. 2 is a flow chart showing an example of a flow of a process to becarried out by a control section of the translation display device.

FIG. 3 is a sequence diagram showing an example of a flow of atranslation display process.

FIG. 4 is a block diagram illustrating main configurations of atranslation display device in accordance with Embodiment 2 of thepresent invention.

FIG. 5 is a flow chart showing an example of a flow of a process to becarried out by a control section of the translation display device.

DESCRIPTION OF EMBODIMENTS Embodiment 1

The following description will discuss an embodiment of the presentinvention with reference to FIGS. 1 through 3.

FIG. 1 is a block diagram illustrating a main configurations of atranslation display device 1. As illustrated in FIG. 1, the translationdisplay device 1 includes a camera 2, a control section 3, an OCRprocessing section 4, a translation processing section (translatingsection) 5, a storage section 6, an image movement analyzing section(movement information identifying section) 7, and a display section 8.Examples of the translation display device 1 encompass a smartphone, amobile phone, a digital camera, a digital video camera, a car navigationsystem, a personal computer (PC), and a personal digital assistant(PDA).

The camera 2 captures an image of an object such as text. For example,the camera 2 captures several to several tens of images per second andtransmits the images thus captured to a camera image obtaining section(image obtaining section) 31.

The control section 3 (i) carries out various types of calculations byexecuting a program which has been supplied from the storage section 6to a temporary storage section (not illustrated) and (ii)comprehensively controls sections included in the translation displaydevice 1. According to Embodiment 1, the control section 3 includes, asfunction blocks, the camera image obtaining section 31, anOCR/translation instructing section 32, a movement analysis instructingsection 33, and an image generating section 34.

The camera image obtaining section 31 sequentially obtains capturedimages captured by the camera 2 and then transmits the captured imagesto the OCR/translation instructing section 32 and the movement analysisinstructing section 33.

The OCR/translation instructing section 32 receives a captured imagefrom the camera image obtaining section 31 and then instructs the OCRprocessing section 4 and the translation processing section 5 to subjectthe captured image to an OCR process and a translation process,respectively. Specifically, the OCR/translation instructing section 32first transmits, to the OCR processing section 4, the received capturedimage serving as an instruction to start an OCR process so that thereceived captured image is subjected to the OCR process. Note that thecaptured image that the OCR/translation instructing section 32 hastransmitted to the OCR processing section 4 will be referred to as atranslation image. Then, the OCR/translation instructing section 32receives, from the translation processing section 5, translated text andtext position data which serve as a process end report. The translatedtext is text which is obtained by translating text extracted from thetranslation image. The text position data is information indicative of aposition of the text extracted from the translation image. In a casewhere the OCR/translation instructing section 32 has received thetranslated text and the text position data, the OCR/translationinstructing section 32 controls the storage section 6 to storetranslation information in which the translation image, the translatedtext, and the text position data are associated with each other. In acase where the OCR/translation instructing section 32 has received theprocess end report, the OCR/translation instructing section 32transmits, to the movement analysis instructing section 33, imageidentification information (an image ID or the like) indicative of thetranslation image which has been subjected to the OCR process and thetranslation process. Note that a plurality of pieces of translated textcan be associated with a translation image. In such a case, a pluralityof pieces of text position data corresponding to the respectiveplurality of pieces of translated text are associated with thetranslation image. As described later, the translation processingsection 5 subjects, to a translation process, text which has beenextracted by the OCR processing section 4. According to Embodiment 1,therefore, the OCR/translation instructing section 32 does not transmita specific instruction to the translation processing section 5.

Note, however, that the OCR/translation instructing section 32 instructsthe OCR processing section 4 and the translation processing section 5 tocarry out respective processes in a case where the OCR processingsection 4 or the translation processing section 5 is not subjecting, toa process, another image which differs from the captured image that theOCR/translation instructing section 32 has received. For example, duringa period from a time point at which the OCR/translation instructingsection 32 instructs the OCR processing section 4 to start an OCRprocess to a time point at which the OCR/translation instructing section32 receives a process end report from the translation processing section5, the OCR/translation instructing section 32 can judge that the OCRprocessing section 4 or the translation processing section 5 is carryingout a process. Note that in a case where the OCR/translation instructingsection 32 receives a captured image from the camera image obtainingsection 31 while the OCR processing section 4 or the translationprocessing section 5 is carrying out a process, the OCR/translationinstructing section 32 does not instruct the OCR processing section 4 tosubject the captured image thus received to an OCR process.

In a case where the movement analysis instructing section 33 hasreceived a captured image from the camera image obtaining section 31,the movement analysis instructing section 33 instructs the imagemovement analyzing section 7 to analyze an image movement in thecaptured image thus received. Note that the captured image that themovement analysis instructing section 33 has instructed the imagemovement analyzing section 7 to analyze will be referred to as amovement analysis image. To describe the movement analysis instructingsection 33 specifically, the movement analysis instructing section 33identifies, as a reference image to be used for comparison with themovement analysis image, an image indicated by image identificationinformation which has been received from the OCR/translation instructingsection 32 most recently. The movement analysis instructing section 33then instructs the image movement analyzing section 7 to analyze themovement analysis image with respect to the reference image. Morespecifically, the movement analysis instructing section 33 transmits, tothe image movement analyzing section 7, (i) the captured image and (ii)the image identification information indicative of the reference imagewhich have been received. The captured image and the imageidentification information thus transmitted serve as an instruction toanalyze an image movement.

In a case where the image generating section 34 has received movementinformation and a movement analysis image from the image movementanalyzing section 7, the image generating section 34 reads, from thestorage section 6, translated text of and text position data of areference image which has been compared with the movement analysis imagethus received. In accordance with the text position data and themovement information, the image generating section 34 identifies aposition (a translation display position) at which the translated textis to be displayed in the movement analysis image thus received. Theimage generating section 34 generates an image (a generated image) bysuperimposing the translated text on the translation display position ofthe movement analysis image. The image generating section 34 transmitsthe generated image to the display section 8 and controls the displaysection 8 to display the generated image.

The OCR processing section 4 receives a translation image from theOCR/translation instructing section 32, extracts text from thetranslation image, and then generates OCR text indicative of the textthus extracted. The OCR processing section 4 generates, as text positiondata, data indicative of a position from which the text has beenextracted. The OCR processing section 4 transmits, to the translationprocessing section 5, the OCR text and the text position data which havebeen thus generated.

In a case where the translation processing section 5 has received OCRtext and text position data from the OCR processing section 4, thetranslation processing section 5 carries out a translation process.Specifically, the translation processing section 5 translates the OCRtext thus received into a language other than a language used in the OCRtext. Alternatively, the OCR text can be translated into a plurality oflanguages. In a case where the translation processing section 5 hasfinished translating the OCR text, the translation processing section 5transmits, to the OCR/translation instructing section 32, translatedtext and the text position data which serve as a process end report.

The storage section 6 stores a translation image, translated text, andtext position data so that the translation image, the translated text,and the text position data are associated with each other.

In a case where the image movement analyzing section 7 has received,from the movement analysis instructing section 33, a movement analysisimage and image identification information which serve as an instructionto analyze an image movement, the image movement analyzing section 7reads, from the storage section 6, a translation image which isindicated by the image identification information and which serves as areference image. The image movement analyzing section 7 analyzes adifference between the reference image and the movement analysis imageso as to generate movement information. Then, the image movementanalyzing section 7 transmits, to the image generating section 34, (i)the movement information thus generated and (ii) the movement analysisimage which has been analyzed. Note that the movement informationindicates a displacement of a position of an object in a movementanalysis image from a position of an object in a reference image. Forexample, the movement information can be information (movement vectorinformation) indicative of (i) a direction in which an object has movedand (ii) an amount by which the object has moved. In addition, themovement information can include information indicative of a differencein composition and zoom between the reference image and the movementanalysis image.

The image movement analyzing section 7 can generate movement informationby comparing an entire portion of the reference image with an entireportion of the movement analysis image. Alternatively, the imagemovement analyzing section 7 can generate movement information bycomparing regions of the reference image with the corresponding regionsof the movement analysis image. This makes it possible to generate,region by region, movement information indicative of a displacement ofthe movement analysis image. Since movement information is individuallygenerated region by region, movements of objects belonging to respectiveregions of an image can be reflected in corresponding pieces of themovement information even in a case where, for example, the objects havemoved in differing directions by differing amounts.

The display section 8 displays an image in accordance with aninstruction provided by the control section 3. It is only necessary thatthe display section 8 display an image in accordance with an instructionprovided by the control section 3. Examples of displays which can beapplied to the display section encompass a liquid crystal display (LCD),an organic EL display, and a plasma display.

A flow of a process to be carried out by the control section 3 of thetranslation display device 1 in accordance with Embodiment 1 of thepresent invention will be described next. FIG. 2 is a flow chart showingan example of the flow of the process to be carried out by the controlsection 3 of the translation display device 1.

As illustrated in FIG. 2, the camera image obtaining section 31 obtainsa captured image captured by the camera 2 and then transmits thecaptured image to the OCR/translation instructing section 32 and to themovement analysis instructing section 33 (step S1: image obtainingstep). In a case where the OCR/translation instructing section 32 hasreceived the captured image, the OCR/translation instructing section 32judges whether or not the OCR processing section 4 or the translationprocessing section 5 is carrying out a process (step S2). In a casewhere the OCR/translation instructing section 32 has judged that the OCRprocessing section 4 or the translation processing section 5 is notcarrying out a process (No in step S2), the OCR/translation instructingsection 32 transmits, as a translation image to be translated, thereceived captured image to the OCR processing section 4 so as toinstruct the OCR processing section 4 to subject the translation imageto the OCR process (step S3). Then, the OCR/translation instructingsection 32 receives, from the translation processing section 5,translated text and text position data which serve as a process endreport (step S4). Then, the OCR/translation instructing section 32controls the storage section 6 to store the text position data, thetranslated text, and the translation image so that the text positiondata, the translated text, and the translation image are associated witheach other. The OCR/translation instructing section 32 also transmits,to the movement analysis instructing section 33, information indicativeof the translation image thus stored (step S5). Then, the camera imageobtaining section 31 judges whether or not there exists an input image(step S6). In a case where the camera image obtaining section 31 hasjudged that there exists no input image (No in step S6), the processcarried out by the control section 3 ends. In a case where the cameraimage obtaining section 31 has judged that there exists an input image(Yes in step S6), the process carried out by the control section 3proceeds to the step S1. Note that in a case where the OCR/translationinstructing section 32 has judged that the OCR processing section 4 orthe translation processing section 5 is carrying out a process onanother image (in a case of Yes in the step S2), the process carried outby the control section 3 proceeds to the step S6.

In a case where the camera image obtaining section 31 has received acaptured image (step S1), the control section 3 carries out steps S7through 10 in parallel with the steps S2 through 6. The steps S7 through10 will be described below. In a case where the movement analysisinstructing section 33 has received a captured image from the cameraimage obtaining section 31, the movement analysis instructing section 33instructs the image movement analyzing section 7 to analyze the capturedimage thus received (step S7). In a case where the image generatingsection 34 has received, from the image movement analyzing section 7,(i) a movement analysis image, of which the movement analysisinstructing section 33 has instructed a movement analysis and (ii)movement information on the movement analysis image, the imagegenerating section 34 reads, from the storage section 6, translated textof and text position data of a reference image. Then, in accordance with(i) a position indicated by the text position data and (ii) the movementinformation, the image generating section 34 identifies a displayposition at which the translated text is to be displayed in the movementanalysis image. Then, the image generating section 34 generates agenerated image in which the translated text is superimposed on thedisplay position of the movement analysis image (step S8). The imagegenerating section 34 transmits the generated image to the displaysection 8 and then controls the display section 8 to display thegenerated image (step S9: display step). Then, the camera imageobtaining section 31 judges whether or not there exists an input image(step S10). In a case where the camera image obtaining section 31 hasjudged that there exists no input image (No in step S10), the processcarried out by the control section 3 ends. In a case where the cameraimage obtaining section 31 has judged that there exists an input image(Yes in step S10), the process carried out by the control section 3proceeds to the step S1.

A specific flow of a translation display process to be carried out bythe translation display device 1 in accordance with Embodiment 1 of thepresent invention will be described next. FIG. 3 is a sequence diagramshowing an example of the specific flow of the translation displayprocess. In the example, it is assumed that (i) an image P0 was alreadysubjected to an OCR/translation process before the translation displayprocess illustrated in FIG. 3 is started, (ii) the image P0, translatedtext of the image P0, and text position data of the image P0 are storedin the storage section 6, and (iii) the image P0 is a reference image ata time point at which the translation display process starts.

As illustrated in FIG. 3, the control section 3 obtains an image P1(step S20: an image obtaining step). Neither the OCR processing section4 nor the translation processing section 5 is carrying out a processwhen the control section 3 obtains the image P1. Therefore, the controlsection 3 provides the OCR processing section 4 with an instruction tosubject the image P1 to an OCR process (step S21). In response to theinstruction from the control section 3, the OCR processing section 4generates OCR text and text position data from the image P1, and thentransmits the OCR text and the text position data to the translationprocessing section 5 (step S22). In a case where the translationprocessing section 5 has received the OCR text and the text positiondata, the translation processing section 5 subjects the OCR text to atranslation process so as to generate translated text (step S23: atranslation step). The translation processing section 5 then transmits,to the control section 3, the translated text and the text position datawhich serve as a process end report (step S24).

In the case where the image P1 has been supplied to the control section3 (step S20), the control section 3 provides, in parallel with the stepS21 being carried out, the image movement analyzing section 7 with aninstruction to subject the image P1 to a movement analysis process byusing the image P0 as a reference (step S25). In response to theinstruction from the control section 3, the image movement analyzingsection 7 identifies movement information which indicates a displacementof a position of an object in the image P1 with respect to a referencewhich is a position of an object in the image P0 (step S26: a movementinformation identifying step). The image movement analyzing section 7then transmits, to the control section 3, the movement information thusgenerated and the image P1 (step S27). In a case where the controlsection 3 has received the image P1 and the movement information of theimage P1 from the image movement analyzing section 7, the controlsection 3 reads, from the storage section 6, translated text of and textposition data of the image P0 which, as a reference image, has beencompared with the image P1. In accordance with (i) a position indicatedby the text position data and (ii) the movement information, the controlsection 3 identifies a position at which translated text is to bedisplayed in the image P1. The control section 3 then (i) generates agenerated image in which the translated text is superimposed on theposition thus identified in the image P1 and (ii) controls the displaysection 8 to display the generated image (step S28: display step).

An image P2 is next supplied to the control section 3 so as to followthe image P1 (step S20 a). Since the OCR processing section 4 isprocessing the image P1, the control section 3 does not instruct the OCRprocessing section 4 to subject the image P2 to an OCR process.Meanwhile, the control section 3 provides the image movement analyzingsection 7 with an instruction to subject the image P2 to a movementanalysis process (step S25 a). The image P0 is an image which was mostrecently translated. Therefore, in response to the instruction from thecontrol section 3, the image movement analyzing section 7 identifiesmovement information of the image P2 with respect to the image P0serving as a reference (step S26 a). The image movement analyzingsection 7 then transmits, to the control section 3, the image P2 and themovement information of the image P2 (step S27 a). In a case where thecontrol section 3 has received the movement information and the image P2from the image movement analyzing section 7, the control section 3reads, from the storage section 6, translated text of and text positiondata of the reference image P0. The control section 3 generates agenerated image in which translated text of the reference image P0 issuperimposed on a translation display position of the image P2, whichtranslation display position has been identified in accordance with (i)the text position data of the reference image P0 and (ii) the movementinformation on the image P2. The control section 3 then controls thedisplay section 8 to display the generated image (step S28 a).

An image P3 is next supplied to the control section 3 so as to followthe image P2 (step S20 b). Since the translation processing section 5 isprocessing the image P1, the control section 3 does not provide the OCRprocessing section 4 with an instruction to subject the image P3 to anOCR process. Meanwhile, the control section 3 provides the imagemovement analyzing section 7 with an instruction to subject the image P3to a movement analysis process (step S25 b). In subsequent steps S26 b,27 b, and 28 b, the image P3 is processed as with the image P2 in thesteps S26 a, 27 a, and 28 a. Therefore, descriptions of the steps S26 b,27 b, and 28 b will be omitted.

An image P4 is next supplied to the control section 3 (step S20 c). Notethat before the control section 3 obtains the image P4, the controlsection 3 has received a translation process end report on the image P1.The control section 3 therefore (i) judges that neither the OCRprocessing section nor the translation processing section is carryingout a process and therefore (ii) provides the OCR processing section 4with an instruction to subject the image P4 to an OCR process (step S21c). In response to the instruction from the control section 3, the OCRprocessing section 4 subjects the image P4 to a process similar to thatcarried out in the step S22 (step S22 a).

In the case where the image P4 has been supplied to the control section3 (step S20 c), the control section 3 provides, in parallel with thestep S21 c being carried out, the image movement analyzing section 7with an instruction subject the image P4 to a movement analysis process(step S25 c). In so doing, since the image P1 is an image which was mostrecently translated, the control section 3 provides the image movementanalyzing section 7 with an instruction to analyze an image movement inthe image P4 by using the image P1 as a reference. In response to theinstruction from the control section 3, the image movement analyzingsection 7 generates movement information of the image P4 by using thereference image P1 as a reference (step S26 c). The image movementanalyzing section 7 transmits, to the control section 3, (i) themovement information thus generated and (ii) the image P4 (step S27 c).In a case where the control section 3 has received the image P4 and themovement information of the image P4 from the image movement analyzingsection 7, the control section 3 reads, from the storage section 6,translated text of and text position data of the reference image P1which has been compared with the image P4. The control section 3generates a generated image in which the translated text of thereference image P1 is superimposed on a translation display position ofthe image P4, which translation display position has been identified inaccordance with (i) the text position data of the reference image P1 and(ii) the movement information on the image P4. The control section 3then controls the display section 8 to display the generated image (stepS28 c).

As described above, the OCR/translation instructing section 32 controlsthe OCR processing section 4 and the translation processing section 5 tosubject, to an OCR process and a translation process, respectively, animage that the camera image obtaining section 31 first obtained after animage is subjected to an OCR process and to a translation process.Therefore, even in a case where it takes time to carry out an OCRprocess and/or a translation process, it is still possible to output alatest translation result in accordance with the time it took to carryout the OCR process and/or the translation process. That is, it ispossible to maintain, at a certain level or more, each of (i) accuracyof a position of text which has been extracted from an image and whichis to be translated and (ii) accuracy of a translation of the text.

In accordance with (i) a position, in a reference image, from which texthas been extracted and (ii) movement information which indicates adisplacement of a position of text in a captured image with respect tothe position of the text in the reference image, the image generatingsection 34 identifies a position, in the captured image, at whichtranslated text of the reference image is to be displayed. Therefore,the translated text of the reference image can be displayed on thecaptured image so as to follow a proper position of the reference imagewithout delaying a timing at which an image is to be displayed (i.e.,without reducing a frame rate).

The translation display device 1 can therefore display, without delayinga timing at which an image is to be displayed (i.e., without reducing aframe rate), an image in which a certain level or more of accuracy ismaintained in term of (i) a translation of text and (ii) a position atwhich translated text is to be displayed.

Embodiment 2

The following description will discuss another embodiment of the presentinvention with reference to FIGS. 4 and 5. For convenience, membershaving functions identical to those of the members described inEmbodiment 1 will be given the same reference signs, and descriptions ofsuch members will be omitted. With reference to a function block diagramillustrated in FIG. 4, a configuration of a translation display device 1a in accordance with Embodiment 2 will be described below.

As illustrated in FIG. 4, the translation display device 1 a includes asensor (detection section) 9 in addition to the members included in thetranslation display device 1 of Embodiment 1. The translation displaydevice 1 a includes a control section 3 a instead of including thecontrol section 3.

The control section 3 a includes a sensor information obtaining section35 in addition to the members included in the control section 3. Thecontrol section 3 a further includes a movement analysis instructingsection 33 a and an image generating section 34 a instead of includingthe movement analysis instructing section 33 and the image generatingsection 34, respectively, of the control section 3 of Embodiment 1.

The sensor 9 detects a movement of a camera 2 (e.g., a direction ofmovement, an orientation, a rotation, a distance of movement, a speed ofmovement, etc.). The sensor 9 can be, for example, a six-axis sensor orthe like. The sensor 9 transmits, to a sensor information obtainingsection 35, a camera movement detection value indicating the movement ofthe camera 2.

In a case where the sensor information obtaining section 35 obtains thecamera movement detection value from the sensor 9, the sensorinformation obtaining section 35 transmits the camera movement detectionvalue to the movement analysis instructing section 33 a.

In addition to the configuration of the movement analysis instructingsection 33, the movement analysis instructing section 33 a includes thefollowing configuration. In a case where the movement analysisinstructing section 33 a receives a captured image from a camera imageobtaining section 31, the movement analysis instructing section 33 amakes a judgment regarding whether or not a value concerning themovement of the camera 2 has exceeded a predetermined range.Specifically, the movement analysis instructing section 33 a makes thejudgment in accordance with the camera movement detection value withrespect to a reference state of the camera 2 at a time point at which areference image was captured (a reference time point). For example, theabove judgement can be made by referring to camera movement detectionvalues obtained (i) at the reference time point and (ii) at a time pointat which a captured image was captured. In a case where the movementanalysis instructing section 33 a has judged that the value concerningthe movement of the camera 2 has exceeded the predetermined range, themovement analysis instructing section 33 a does not instruct the imagemovement analyzing section 7 to carry out a movement analysis processand transmits, to the image generating section 34 a, the captured imagereceived from the camera image obtaining section 31.

Examples of the predetermined range with respect to the value concerningthe movement of the camera 2 encompass (i) a predetermined range ofangles within which the camera 2 rotates, (ii) a predetermined range ofspeeds within which the camera 2 moves, and (iii) a predetermined rangeof distances within which the camera 2 moves. More detailed examples ofa case where the value concerning the movement of the camera 2 exceeds apredetermined range encompass (i) a case where an orientation of thecamera has been rotated by 30 degrees or more from an orientation at thereference time point, (ii) a case where the camera has moved at a speedof 10 cm/second or more, and (iii) a case where the camera has moved bya predetermined distance or more from a position measured at thereference time point.

The image generating section 34 a includes the configuration of theimage generating section 34 described above. In a case where the imagegenerating section 34 a receives the captured image from the movementanalysis instructing section 33 a, the image generating section 34 atransmits the captured image to a display section 8 and then controlsthe display section 8 to display the captured image while the capturedimage has not been subjected to any process after being obtained.

In a case where a value concerning the movement of the camera 2 exceedsthe predetermined range at a time point at which a captured image iscaptured by the camera 2, text extracted from the reference image maynot have been captured in the captured image. In such a case, it isunnecessary to follow a position of extracted text or to displaytranslated text of the extracted text. According to the presentinvention, in a case where a value concerning a movement of the camera 2exceeds a predetermined range, the image movement analyzing section 7does not identify movement information, and the display section 8displays an image obtained by the camera image obtaining section 31while the image has not been subjected to any process after beingobtained. This means that an unnecessary process is not carried out, andthat it is therefore possible to reduce a processing load of thetranslation display device.

Note that in a case where (i) a value concerning a movement of thecamera 2 once exceeds a predetermined range and (ii) the valueconcerning the movement of the camera 2 falls within the predeterminedrange at a time point at which a received captured image is captured,the movement analysis instructing section 33 a judges that the valueconcerning the movement of the camera 2 has not exceeded thepredetermined range. For example, in a case where the camera 2 rotatesby 30 degrees or more from an orientation measured at a reference timepoint and then returns to the orientation, the movement analysisinstructing section 33 a judges that the value concerning the movementof the camera 2 falls within the predetermined range so that themovement analysis instructing section 33 a instructs the image movementanalyzing section 7 to carry out an image movement analysis. Therefore,in a case where text once falls outside a capturing range of the cameraand then returns into the capturing range of the camera, it is possibleto display translated text of text so that the translated text follows aposition of the text.

A flow of a process to be carried out by the control section 3 a of thetranslation display device 1 a in accordance with Embodiment 2 of thepresent invention will be described next. FIG. 5 is a flow chart showingan example of the flow of the process to be carried out by the controlsection 3 a. Part of the flow of the process, which part is similar tothat of the process of the control section 3 in accordance withEmbodiment 1, will not be described below.

Subsequently to the step S1, the movement analysis instructing section33 a receives a captured image from the camera image obtaining section31. Then, the movement analysis instructing section 33 a judges whetheror not a value concerning a movement of the camera 2 from a referencetime point at which a reference image stored in a storage section 6 hadbeen captured to a time point at which the received captured image hasbeen captured exceeds a predetermined range (step S11). In a case wherethe movement analysis instructing section 33 a has judged that the valueconcerning the movement of the camera 2 exceeds the predetermined range(Yes in step S11), the movement analysis instructing section 33 atransmits, to the image generating section 34 a, the captured imagereceived from the camera image obtaining section 31 (step S12). A stepS9 subsequent to the step S12 is carried out in the following manner.The image generating section 34 a treats, as a generated image withoutany processing, the captured image received from the movement analysisinstructing section 33 a. The image generating section 34 a transmitsthe generated image to the display section 8 and then controls thedisplay section 8 to display the generated image (step S9). The processcarried out by the control section 3 a then proceeds to a step S10.

That is, according to the translation display device 1 a, the cameraimage obtaining section 31 obtains an image captured by the camera 2,and the translation display device 1 a further includes the sensor 9 fordetecting a movement of the camera 2. In a case where a value concerningthe movement of the camera 2 with respect to a state of the camera at atime point at which the reference image was captured exceeds thepredetermined range, the image movement analyzing section 7 does notidentify movement information. The display section 8 displays the imageobtained by the camera image obtaining section 31 while the image hasnot been subjected to any process after being obtained.

In a case where the value concerning the movement of the camera 2exceeds a predetermined range, text extracted from a reference image maynot be captured in an image that the camera image obtaining section 31has most recently obtained. In such a case, it is unnecessary to followa position of extracted text or to display translated text of theextracted text.

According to the configuration, in a case where the value concerning themovement of the camera 2 exceeds the predetermined range, the imagemovement analyzing section 7 does not identify movement information, andthe display section 8 displays an image obtained by the camera imageobtaining section 31 while the image has not been subjected to anyprocess after being obtained. This means that an unnecessary process isnot carried out, and it is therefore possible to reduce a processingload of the translation display device.

According to the configuration, in a case where the value concerning themovement of the camera 2 exceeds the predetermined range at a currenttime point (a time point at which an image most recently obtained by thecamera image obtaining section 31 was captured), no unnecessary processis carried out. Therefore, in a case where (i) the value concerning themovement of the camera 2 has once exceeded the predetermined range and(ii) the value concerning the movement of the camera 2 at a current timepoint falls within the predetermined range, the image movement analyzingsection 7 identifies movement information and the display section 8displays translated text so that the translated text is superimposed onthe image that has been most recently obtained. Therefore, in a casewhere text once falls outside a capturing range of the camera and thenthe text returns into the capturing range of the camera, it is possibleto display translated text of text so that the translated text follows aposition of the text.

Embodiment 3

The control section 3, the OCR processing section 4, the translationprocessing section 5, and the image movement analyzing section 7 of thetranslation display device 1 and the control section 3 a, the OCRprocessing section 4, the translation processing section 5, and theimage movement analyzing section 7 of the translation display device 1 acan be realized by a logic circuit (hardware) provided in an integratedcircuit (IC chip) or the like or can be alternatively realized bysoftware as executed by a central processing unit (CPU).

In the latter case, the translation display devices 1 and 1 a eachinclude a CPU that executes instructions of a program that is softwarerealizing the foregoing functions; a read only memory (ROM) or a storagedevice (each referred to as “storage medium”) in which the program andvarious kinds of data are stored so as to be readable by a computer (ora CPU); and a random access memory (RAM) in which the program is loaded.An object of the present invention can be achieved by a computer (or aCPU) reading and executing the program stored in the storage medium.Examples of the storage medium encompass “a non-transitory tangiblemedium” such as a tape, a disk, a card, a semiconductor memory, and aprogrammable logic circuit. The program can be supplied to the computervia any transmission medium (such as a communication network or abroadcast wave) which allows the program to be transmitted. Note thatthe present invention can also be achieved in the form of a computerdata signal in which the program is embodied via electronic transmissionand which is embedded in a carrier wave.

SUMMARY

A translation display device (1, 1 a) in accordance with a first aspectof the present invention is a translation display device including: animage obtaining section (image obtaining section 31) for sequentiallyobtaining images; a translating section (translation processing section5) for a carrying out a translation process of translating textextracted from any given image, the translating section subjecting acertain image to the translation process in a case where the imageobtaining section obtains the certain image while no image other thanthe certain image is being subjected to the translation process; amovement information identifying section (image movement analyzingsection 7) for identifying movement information indicative of adisplacement of a position of an object in a most recent image which hasbeen most recently obtained by the image obtaining section, thedisplacement being measured with respect to a reference position of anobject in a reference image for which the translation process mostrecently ended; and a display section (display section 8) for displayinga generated image in which translated text obtained by translatingextracted text extracted from the reference image is superimposed on themost recent image in accordance with (i) a position of the extractedtext and (ii) the movement information.

According to the configuration, the translating section carries out atranslation process of translating text extracted from a certain imageobtained by the image obtaining section, in a case where no image otherthan the certain image is being subjected to the translation process,whereas no translation process is carried out with respect to a certainimage obtained by the image obtaining section, in a case where anotherimage other than the certain image is being subjected to the translationprocess. In other words, the translating section subjects, to atranslation process, an image that the image obtaining section firstobtained after an image is subjected to a translation process.Therefore, even in a case where it takes time to carry out a translationprocess, it is still possible for the translating section to output alatest translation result in accordance with the time it took to carryout the translation process. That is, it is possible to maintain, at acertain level or more, each of (i) accuracy of a position of text whichhas been extracted from an image and which is to be translated and (ii)accuracy of a translation of the text.

The display section displays a generated image in which translated textobtained by translating extracted text extracted from the referenceimage is superimposed on the most recent image in accordance with (i) aposition of the extracted text and (ii) the movement information. Thatis, the display section displays, in accordance with a translationresult which has been obtained in the past, translated text of text sothat the translated text follows a position of the text. This allows thedisplay section to display translated text so that the translated textis superimposed on an image at a timing at which the image obtainingsection obtains the image. The display section displays the translatedtext in accordance with a latest translation result and, as describedabove, it is therefore possible to display the translated text whilemaintaining, at a certain level or more, each of (i) accuracy of atranslation of the translated text and (ii) accuracy of a position atwhich the translated text is to be displayed.

The translation display device brings about an effect of displaying,without delaying a timing at which an image is to be displayed (i.e.,without reducing a frame rate), an image in which (i) translation oftext and (ii) a position at which translated text is to be displayedmaintain accuracy at a certain level or more.

In a second aspect of the present invention, a translation displaydevice (1 a) can further include, in the first aspect of the presentinvention, a detection section (sensor 9) for detecting a movement of acamera (2), the image obtaining section obtaining an image captured bythe camera, in a case where a value concerning the movement of thecamera with respect to a reference state of the camera at a time pointat which the reference image was captured by the camera exceeds apredetermined range, (i) the movement information identifying sectionnot identifying movement information and (ii) the display sectiondisplaying an image obtained by the image obtaining section while theimage has not been subjected to any process after being obtained.

In a case where a value concerning a movement of the camera exceeds apredetermined range, text extracted from the reference image may nothave been captured in an image which has been most recently obtained bythe image obtaining section. In such a case, it is unnecessary to followa position of extracted text or to display translated text of the text.

According to the configuration, in a case where the value concerning themovement of the camera exceeds the predetermined range, the movementinformation identifying section does not identify movement information,and the display section displays an image obtained by the imageobtaining section while the image has not been subjected to any processafter being obtained. This means that an unnecessary process is notcarried out, and it is therefore possible to reduce a processing load ofthe translation display device.

According to the configuration, in a case where the value concerning themovement of the camera exceeds the predetermined range at a current timepoint (a time point at which an image most recently obtained by theimage obtaining section was captured), no unnecessary process is carriedout. Therefore, in a case where (i) the value concerning the movement ofthe camera has once exceeded the predetermined range and (ii) the valueconcerning the movement of the camera at a current time point fallswithin the predetermined range, the movement information identifyingsection identifies movement information and the display section displaystranslated text so that the translated text is superimposed on the imagethat has been most recently obtained. Therefore, in a case where textonce falls outside a capturing range of the camera and then the textreturns into the capturing range of the camera, it is possible todisplay translated text of text so that the translated text follows aposition of the text.

In a third aspect of the present invention, a translation display device(1 a) can be arranged such that, in the second aspect of the presentinvention, the value concerning the movement of the camera with respectto the reference state of the camera at the time point at which thereference image was captured is any one of (i) an angle by which thecamera has rotated, (ii) a speed at which the camera has moved, and(iii) a distance by which the camera has moved.

A method of displaying a translation in accordance with a fourth aspectof the present invention is a method of displaying a translation,including the steps of: (a) sequentially obtaining images; (b) carryingout a translation process of translating text extracted from any givenimage, the step (b) being carried out so that a certain image issubjected to the translation process in a case where the certain imageis obtained while no image other than the certain image is beingsubjected to the translation process; (c) identifying movementinformation indicative of a displacement of a position of an object in amost recent image which has been most recently obtained in the step (a),the displacement being measured with respect to a reference position ofan object in a reference image for which the translation process mostrecently ended in the step (b); and (d) displaying a generated image inwhich translated text obtained by translating extracted text extractedfrom the reference image is superimposed on the most recent image inaccordance with (i) a position of the extracted text and (ii) themovement information. The method brings about an effect similar to thatof the first aspect.

The translation display device in accordance with the aspects of thepresent invention can be realized by a computer. In such a case, thescope of the present invention also encompasses (i) a control programfor the translation display device which program controls thetranslation display device to be realized by the computer throughcontrolling the computer to serve as each of the members included in thetranslation display device and (ii) a computer-readable storage mediumin which the program is stored.

The present invention is not limited to the embodiments, but can bealtered by a skilled person in the art within the scope of the claims.An embodiment derived from a proper combination of technical means eachdisclosed in a different embodiment is also encompassed in the technicalscope of the present invention. Further, it is possible to form a newtechnical feature by combining the technical means disclosed in therespective embodiments.

INDUSTRIAL APPLICABILITY

The present invention is applicable to a smartphone, a mobile phone, adigital camera, a digital video camera, a car navigation system, a PC, aPDA, or the like.

REFERENCE SIGNS LIST

-   -   1, 1 a: Translation display device    -   2 Camera;    -   5: Translation processing section (translating section)    -   7 Image movement analyzing section (movement information        identifying section)    -   8 Display section    -   9 Sensor (detection section)    -   31 Camera image obtaining section (image obtaining section)    -   P0, P1 Reference image    -   P1, P2, P3, P4 Image    -   S1, S20 Image obtaining step    -   S23 Translation step    -   S26 Movement information identifying step    -   S9 Display step

The invention claimed is:
 1. A translation display device comprising: aprocessor and an associated memory; wherein the processor: sequentiallyobtains images; performs a translation process of translating textextracted from any given image, and subjects a certain image to thetranslation process in a case where the certain image is obtained whileno image other than the certain image is being subjected to thetranslation process; and identifies movement information indicative of adisplacement of a position of an object in a most recent image which hasbeen most recently obtained, the displacement being measured withrespect to a reference position of an object in a reference image forwhich the translation process most recently ended; and the translationdisplay device further includes a display that displays a generatedimage in which translated text obtained by translating extracted textextracted from the reference image is superimposed on the most recentimage in accordance with (i) a position of the extracted text and (ii)the movement information, and a sensor that detects a movement of acamera, wherein the processor obtains an image captured by the camera,and in a case where a value concerning the movement of the camera withrespect to a reference state of the camera at a time point at which thereference image was captured by the camera exceeds a predeterminedrange, (i) the processor does not identify movement information and (ii)the display displays the image obtained by the processor while the imagehas not been subjected to any process after being obtained.
 2. Thetranslation display device as set forth in claim 1, wherein the valueconcerning the movement of the camera with respect to the referencestate of the camera at the time point at which the reference image wascaptured is any one of (i) an angle by which the camera has rotated,(ii) a speed at which the camera has moved, and (iii) a distance bywhich the camera has moved.
 3. A method of displaying a translation,comprising the steps of: (a) sequentially obtaining images; (b) carryingout a translation process of translating text extracted from any givenimage, the step (b) being carried out so that a certain image issubjected to the translation process in a case where the certain imageis obtained while no image other than the certain image is beingsubjected to the translation process; (c) identifying movementinformation indicative of a displacement of a position of an object in amost recent image which has been most recently obtained in the step (a),the displacement being measured with respect to a reference position ofan object in a reference image for which the translation process mostrecently ended in the step (b); (d) displaying a generated image inwhich translated text obtained by translating extracted text extractedfrom the reference image is superimposed on the most recent image inaccordance with (i) a position of the extracted text and (ii) themovement information; (e) obtaining a movement of a camera detected by asensor; and (f) obtaining an image captured by the camera, wherein in acase that a value concerning the movement of the camera with respect toa reference state of the camera at a time point at which the referenceimage was captured by the camera exceeds a predetermined range, (i) notidentifying any movement information and (ii) displaying the imageobtained in the step (f) while the image has not been subjected to anyprocess after being obtained.